3.3. Link Layer Interface - PHY Control Signals

The signals in Table 5 shall constitute the control interface from the link layer to the PHY.

Additional implementation-dependent signals may exist.

3.3.1. PHYReset

The PHYReset pin is asserted by the Link layer to initialize the PHY. The PHYReset signal shall reset the internal registers to their HW default states, which shall allow MDIO programming and internal self-alignment to take place.

The reset states are otherwise implementation-dependent and shall be documented in the datasheet of a particular implementation.

3.3.2. MDIO Serial Programming Interface

The MDC and MDIO pins control the programming and status readout of the PHY by the Link layer. Per the spec at https://en.wikipedia.org/wiki/Management_Data_Input/Output up to 32 slices may be connected to one bus and each slice may have up to 32 16-bit registers.

The MDIO bus interface shall operate at up to 25 MHz.

[ Need to add a real MDI spec reference. ]

3.3.3. PHYReady

After the Rx slice clock self-alignments are complete, each Rx PHY slice shall assert its PHYReady pin. These may be AND'd together by the Link layer or treated separately. How an Rx PHY slice determines completion of the self-alignment is implementation-dependent. For instance, it can be determined by observing the settling of the DLL or by a simple timer.

3.3.4. MDIO Register Map

The registers accessed by the MDIO Serial Programming Interface are implementation dependent. The registers shall be fully documented in the datasheet of a particular implementation.

These registers may control:

• Configuration such as Tx vs. Rx, BoW Mode (TxClk rate), PCLK rate
• Test features such as loopback modes
• External control of DCC, DLL
• Readout of status such as DCC and DLL self-alignment settings

4. BoW Modes

4.1. BoW Modes

A BoW interface must conform to one of the Bow Modes seen in Table 6.

The BoW Mode defines the speed of clock and data of the PHY on the die-to-die wires. In all modes, the data must be clocked DDR: the data wire bit rate is double the clock wire frequency. All BoW interfaces must be able to interoperate with all the lower modes. Supporting rates other than the defined four modes is an implementation choice.

The recommended maximum wire reach for different packaging types and terminations is seen in Table 7. Exceeding these reach values degrades the voltage margins at the receiver. “Laminate” is intended to include organic laminate packages (a.k.a. “buildup) and similar technologies with approximately 25 um line and space rules. ”Advanced" is intended to include silicon interposer and similar technologies.
These have much finer line and space dimensions, but traces are usually much more resistive than in organic laminate packages and must operate with reduced trace lengths. Termination is not expected to be necessary for implementations targeting Advanced packaging.

Adding termination increases the speed and/or reach, at the expense of greater design complexity.

5. BoW Physical Configuration

5.1. BoW Components

A BoW link between two chiplets is made up of wires, slices, and stacks as seen in Figure 9.

• The signal traces in the package between chiplets are called wires.
• The basic unit is a slice with 18 or 20 signal bumps. It must have 2 bumps for the differential clock and 16 single-ended data bumps. It may also have the optional single-ended signals AUX and FEC. The long edge of a slice is parallel to the chip edge.
• Multiple slices may be placed in a stack. The slice positions are designated A, B, C, etc, starting with the slice closest to the edge of the chip.
• A link from one chiplet to another must be composed of one or more stacks placed along the chip edge. A link may be configured with equal numbers of Rx and Tx slices, or it may be asymmetric or one-way.

5.2. Example Link

The minimal bidirectional reference link is shown in Figure 10.

In this example, each chiplet has one Tx slice and one Rx slice, arranged in a single stack on each chiplet. In a laminate package, the position-A slices (at the chips' edges) are connected together on the topmost routing layer used for signals and the position-B slices are connected together on the next layer used for signals.

5.3. Die-to-Die Signals

Each BoW slice consists of a differential clock pair, 16 single-ended data wires, and optional wires FEC and AUX. Each BoW slice is unidirectional when in operation. A PHY may be designed as Rx-only and Tx-only slices, or each slice may have both Tx and Rx capability, one of which is selected at configuration time. A bidirectional link is composed of some number of slices configured for Rx and some for Tx.

FEC (Forward Error Correction) is an optional signal that allows using FEC to improve the bit error rate (BER). AUX is an optional signal that can be used for purposes such as DBI, flow control, redundancy, etc. Chiplets A and B will need to agree on the details on FEC and AUX usage, which is defined in the Link layer.

5.4. Bump Arrangements

The reference example in Figure 10 uses hexagonal closest packing for the bumps: two rows for signal bumps and one row for power and ground bumps. In this pattern, the wire pitch is half the bump pitch. In order to maintain the closest bump packing, slices in rows B and D have a different bump pattern than slices A and C. But bump patterns are not specified by BoW; only the signal ordering at the chip edge is specified for interoperability.

Alternate bump arrangements may include:

• 90-degree rotation of the hexagonal packing direction to decrease the wire pitch 14%;
• square bump arrays instead of hexagonal for regularity of layout;
• more than two rows of signal bumps to decrease the wire pitch;
• different ordering of power and ground bumps;
• multiple power and ground rows.

An alternate slice arrangement may be to place the Tx and Rx slices side by side at the chip edges. This would take up more chip edge, but allow all the signals to run on the same package layer.

Somewhat different wire and bump pitches between two chiplets can be accomodated with fan-out in the chip-to-chip wires. This is limited by the max wire length.

5.5. Cross Section

A cross section for an organic laminate (a.k.a. “buildup”) package is shown in Figure 11.

In an organic laminate package, signal layers should be alternated with ground layers in order to maintain a controlled impedance of 50 ohms. Each slice position (A, B, C, D) should be associated with one signal layer and there is no mixing of signals from multiple slices.

In any technology, the position-A slice on chiplet A must be connected to the position-A slice on chiplet B (one must be configured for Tx and one for Rx). The position-B slices are connected together, and so on.

There is no specified limit to the number of slices in a stack. In organic laminate, the practical limit in 2020 is an 8-2-8 laminate which supports 4 slices. Layers on the bottom side of the package typically cannot be used for BoW signals due to low via density passing through the thick central core layer.

In advanced packaging technologies, the shorter wire lengths allow the use of non-controlled-impedance wires and unterminated transmitters and receivers. The smaller wire and space dimensions may allow the wires for multiple slices to be interleaved on a single wiring layer.

5.6. Signal Ordering

A BoW interface must conform to these wire and slice order rules at the edge of the chip:

• The signals for a Tx slice are in the following order at the chip edge, going clockwise around the chiplet: AUX, D0, D1, D2, D3, D4, D5, D6, D7, CLK+, CLK-, D8, D9, D10, D11, D12, D13, D14, D15, FEC
• The signals for an Rx slice are in the reversed order (ascending goes counter-clockwise)
• The Tx slices in a link are numbered from 0 at the upper left edge of the link (facing from the chip center to the edge) and ascending through the Tx slices in a stack, then from stack to stack clockwise.
• The Rx slices in a link are numbered from 0 at the upper right, through the Rx slices in a stack, then stack to stack counterclockwise.

These rules allow BoW chiplets to be connected without signal reordering regardless of chiplet rotations.

5.7. Checkerboard Slice Pattern

For bidirectional links with more than one stack on each side, a checkerboard pattern of Tx and Rx slices should be used (Figure 12). This allows efficient connection of chiplets with differing stack depths and numbers of stacks. Figure 12 shows a bidirectional link with 4 stacks of 4 slices each, for 8 Tx and 8 Rx slices on each chiplet.

An alternate approach may be used: design every slice to operate as either Rx or Tx to be configured after assembly or upon powerup. This allows complete flexibility in interoperability and also provides an opportunity for wafer-test loopback testing.

In BoW-64 at 4 Gb/s/wire, the link in Figure 12 provides a total of 1.0 Tb/s. In an organic substrate using the hexagonal bump pattern of Figure 10 with a bump pitch is 130 um, the total edge width is 4.16 mm without AUX and FEC, or 5.2 mm with; the depth from the edge is 1.35 mm. In an interposer, if the bump pitch is 40 um, the edge width is 1.28 or 1.60 mm and the depth is 0.42 mm.

6. Timing

6.1. Clocking

Figure 13 shows the clock and data flow for a single Tx slice and a single Rx slice. On the Tx side, data bits (and optional FEC and AUX bits) come in a wide word from the link layer, and are serialized to the line rate. At the Rx side, they are sampled with a common slicer clock in most BoW implementations. BoW-256 may optionally implement per-bit delay adjust or per-bit slicer clock adjust.

All BoW interfaces shall be DDR (Double Data Rate) at the chip-to-chip interface: the data bit rate is twice the clock frequency, so data is clocked in on both edges of the clock in the Rx slice.

The following table provides clock and data rates for an example with 4 Gbps wire data rate and M=4 to support a 1 Gbps data rate at the Link-PHY interface.

The ratio M should be limited to integers, preferrably powers of two, and other ratios implemented in a gearbox in the Link layer.

The DDR clock TxClk is provided to the Tx PHY from elsewhere on Chiplet-A. This may come from an on-chip PLL (typically shared across multiple slices) or it can be routed from the RxClk of an Rx slice on Chiplet-A. In order to meet duty cycle requirements, a Duty Cycle Corrector (DCC) may be needed in the Tx slice. TxClk is used to drive the serializers and provide the output CLK+,CLK- to Chiplet-B.

On the Rx side, the PHY must align the slicer clock to sample the data correctly. This may be done with a DLL or adjustable delays or other methods. The PHY shall include control logic to self-align the slicer clock for correct sampling of the data. Alignment is started by signal AlignDll from the Rx Link Layer; the PHY provides a signal DllAligned to the Link Layer when it is complete.

All BoW interfaces shall be source synchronous within a slice. BoW-32 to BoW-128 interfaces do not require per-wire alignment - the signals within a slice are aligned sufficiently well by matching their paths. BoW-256 interfaces may need per-wire delay adjustment or per-slicer clock adjustment.

Clock skew between the slices in each direction of a link depends on the implementation of the TxClk distribution to all the Tx slices. That is, for the data flow from Chiplet A to Chiplet B, the TxClk distribution on Chiplet A sets the the clock skew of the Tx slices on Chiplet A and the clock skew of the Rx slices on Chiplet B, and vice versa for flow from B to A. This skew must be no more than 100 ps/stack along the chip edge. There is no specification of the skew between TxClk on Chiplet A vs TxClk on Chiplet B.

On both the Tx and Rx sides, the Link layer must include a Clock Domain Crossing (CDC) to align the data between CoreClk and PhyClk. These CDCs must also be able to absorb the slice-to-slice clock skew and core clock distribution skew across the whole link.

If DCCs are included in the PHY and they need an alignment cycle, they shall include control logic to perform self-alignment.

6.2. Clock and Data Specifications

6.2.1. Timing Requirements

Figure 14 shows the definition of the eye diagram parameters.

The CLK and data signals at the receiving slice bumps must meet the conditions in Table 10

V_hi of 0.75 V must be supported by all BoW implementations, but other values may be supported.

t_eye must be evaluated for each of the bits in a slice relative to the differential CLK+ - CLK- signal for that slice. t_eye must be evaluated for CLK edges up to 3 UI earlier than the eye center. This is because even though jitter on the data edges is correlated with the CLK jitter at the Tx side, the slicer in the Rx side is likely to use a different CLK edge due to delays in the Rx-side clock alignment circuit (usually a DLL). The evaluation of jitter must include all possible jitter contributors, including reference clock, clock distribution networks, any DCC, PLL and DLL jitter, power-supply noise and switching noise.

The slice to slice clock skew t_skew across the width of a BoW link (along the chip edge) must be less than 100 ps/stack. This is dominated by the TxClk distribution network.

Since these signals do not leave the package, these values must be verified with simulation.

If the slice implementation allows programmatic control of the DLL alignment values, varying those values after locking the DLL may provide timing margin information. If the slice implementation allows programmatic control of the receiver voltage thresholds, varying those values may provide vertical margin information.

The tx_mac_rdy signal shall be forwarded to the link partner, and appropriate status 
and control register shall be updated, in order to inform the Receive link layer
that the Transmit link layeris or is not ready for calibration.